{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Pandas 数据去重",
        "",
        "本教程详细介绍如何在 Pandas 中检测和删除重复数据。",
        "",
        "## 目录",
        "1. 检测重复值",
        "2. 删除重复值",
        "3. 按列去重",
        "4. 高级技巧"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 导入库"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import pandas as pd",
        "import numpy as np"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 1. 检测重复值",
        "",
        "### 方法说明",
        "",
        "`duplicated()` 用于检测重复值。",
        "",
        "**语法:**",
        "```python",
        "df.duplicated(subset=None, keep='first')",
        "```",
        "",
        "**主要参数：**",
        "- `subset`: 指定检查重复的列（默认检查所有列）",
        "- `keep`: 'first' 保留第一个（默认），'last' 保留最后一个，False 标记所有重复",
        "",
        "**返回值：**",
        "- 布尔 Series，True 表示重复",
        "",
        "**特点：**",
        "- ✅ 可以检测完全重复和部分重复",
        "- ✅ 可以指定检查的列",
        "- ✅ 可以选择保留哪个重复值",
        "",
        "**适用场景：** 数据探索阶段，了解重复情况"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例1: 创建包含重复值的数据"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "df = pd.DataFrame({",
        "    'ID': [1, 2, 3, 2, 4, 1, 5],",
        "    '姓名': ['张三', '李四', '王五', '李四', '赵六', '张三', '钱七'],",
        "    '年龄': [25, 30, 35, 30, 28, 25, 32],",
        "    '部门': ['技术', '销售', '技术', '销售', '人事', '技术', '销售']",
        "})",
        "",
        "print(\"原始 DataFrame:\")",
        "print(df)",
        "print(\"\\n数据形状:\", df.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例2: 检测完全重复的行",
        "",
        "检查所有列是否完全相同。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "duplicated_mask = df.duplicated()",
        "print(\"重复值检测结果 (True表示重复):\")",
        "print(duplicated_mask)",
        "print(\"\\n重复行数量:\", duplicated_mask.sum())",
        "print(\"\\n重复的行:\")",
        "print(df[duplicated_mask])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例3: 查看所有重复的行 (keep=False)",
        "",
        "使用 `keep=False` 可以标记所有重复的行（包括第一个）。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "all_duplicates = df.duplicated(keep=False)",
        "print(\"所有重复的行 (包括第一个):\")",
        "print(df[all_duplicates])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例4: 检测特定列的重复",
        "",
        "使用 `subset` 参数指定检查的列。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "print(\"检查 'ID' 列的重复:\")",
        "id_duplicates = df.duplicated(subset=['ID'])",
        "print(df[id_duplicates])",
        "",
        "print(\"\\n检查 '姓名' 列的重复:\")",
        "name_duplicates = df.duplicated(subset=['姓名'])",
        "print(df[name_duplicates])",
        "",
        "print(\"\\n检查 'ID' 和 '姓名' 两列的重复:\")",
        "multi_duplicates = df.duplicated(subset=['ID', '姓名'])",
        "print(df[multi_duplicates])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例5: 统计重复情况"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "print(\"完全重复的行数:\", df.duplicated().sum())",
        "print(\"按 ID 重复的行数:\", df.duplicated(subset=['ID']).sum())",
        "print(\"按姓名重复的行数:\", df.duplicated(subset=['姓名']).sum())",
        "print(\"\\n重复率:\", df.duplicated().sum() / len(df) * 100, \"%\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 2. 删除重复值",
        "",
        "### 方法说明",
        "",
        "`drop_duplicates()` 用于删除重复值。",
        "",
        "**语法:**",
        "```python",
        "df.drop_duplicates(subset=None, keep='first', inplace=False, ignore_index=False)",
        "```",
        "",
        "**主要参数：**",
        "- `subset`: 指定检查重复的列（默认检查所有列）",
        "- `keep`: 'first' 保留第一个（默认），'last' 保留最后一个，False 删除所有重复",
        "- `inplace`: 是否在原 DataFrame 上修改",
        "- `ignore_index`: 是否重新设置索引",
        "",
        "**特点：**",
        "- ✅ 可以删除完全重复和部分重复",
        "- ✅ 可以选择保留哪个重复值",
        "- ✅ 支持按列去重",
        "",
        "**适用场景：** 确认重复值需要删除时"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例1: 删除完全重复的行 (保留第一个)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "df_cleaned = df.drop_duplicates()",
        "print(\"删除完全重复的行 (keep='first'):\")",
        "print(df_cleaned)",
        "print(\"\\n删除前:\", len(df), \"行\")",
        "print(\"删除后:\", len(df_cleaned), \"行\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例2: 保留最后一个重复值"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "df_cleaned = df.drop_duplicates(keep='last')",
        "print(\"删除重复的行，保留最后一个 (keep='last'):\")",
        "print(df_cleaned)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例3: 删除所有重复的行"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "df_cleaned = df.drop_duplicates(keep=False)",
        "print(\"删除所有重复的行 (keep=False):\")",
        "print(df_cleaned)",
        "print(\"\\n删除前:\", len(df), \"行\")",
        "print(\"删除后:\", len(df_cleaned), \"行\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例4: 按特定列去重",
        "",
        "基于指定列删除重复，其他列可能不同。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "print(\"原始数据:\")",
        "print(df)",
        "",
        "print(\"\\n按 'ID' 列去重 (保留第一个):\")",
        "df_id = df.drop_duplicates(subset=['ID'])",
        "print(df_id)",
        "",
        "print(\"\\n按 '姓名' 列去重:\")",
        "df_name = df.drop_duplicates(subset=['姓名'])",
        "print(df_name)",
        "",
        "print(\"\\n按 'ID' 和 '姓名' 两列去重:\")",
        "df_multi = df.drop_duplicates(subset=['ID', '姓名'])",
        "print(df_multi)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例5: 使用 inplace 参数直接修改"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "df_copy = df.copy()",
        "df_copy.drop_duplicates(inplace=True)",
        "print(\"使用 inplace=True 直接修改:\")",
        "print(df_copy)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例6: 重新设置索引"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "df_cleaned = df.drop_duplicates(ignore_index=True)",
        "print(\"删除重复并重新设置索引 (ignore_index=True):\")",
        "print(df_cleaned)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 3. 按列去重的高级应用",
        "",
        "### 应用场景1: 数据更新场景",
        "",
        "有时重复记录中，需要保留最新的数据。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "df_update = pd.DataFrame({",
        "    'ID': [1, 1, 2, 2, 3],",
        "    '姓名': ['张三', '张三', '李四', '李四', '王五'],",
        "    '更新时间': ['2024-01-01', '2024-01-15', '2024-01-05', '2024-01-20', '2024-01-10'],",
        "    '工资': [8000, 8500, 12000, 13000, 15000]",
        "})",
        "",
        "df_update['更新时间'] = pd.to_datetime(df_update['更新时间'])",
        "df_update = df_update.sort_values('更新时间')",
        "",
        "print(\"原始数据 (已按时间排序):\")",
        "print(df_update)",
        "",
        "print(\"\\n按 ID 去重，保留最后一条 (最新):\")",
        "df_latest = df_update.drop_duplicates(subset=['ID'], keep='last')",
        "print(df_latest)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 4. 高级技巧",
        "",
        "### 技巧1: 根据条件选择保留的重复值"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "df_conditional = pd.DataFrame({",
        "    'ID': [1, 1, 2, 2, 3],",
        "    '姓名': ['张三', '张三', '李四', '李四', '王五'],",
        "    '工资': [8000, 8500, 12000, 11000, 15000]",
        "})",
        "",
        "print(\"原始数据:\")",
        "print(df_conditional)",
        "",
        "# 保留工资最高的记录",
        "print(\"\\n保留工资最高的记录:\")",
        "df_max = df_conditional.sort_values('工资', ascending=False).drop_duplicates(subset=['ID'], keep='first')",
        "print(df_max)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 技巧2: 标记重复但不删除",
        "",
        "有时需要知道哪些是重复的，但不删除。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "df = pd.DataFrame({",
        "    'ID': [1, 2, 3, 2, 4, 1],",
        "    '姓名': ['张三', '李四', '王五', '李四', '赵六', '张三'],",
        "    '年龄': [25, 30, 35, 30, 28, 25]",
        "})",
        "",
        "# 添加重复标记列",
        "df['是否重复'] = df.duplicated(subset=['ID'], keep=False)",
        "df['重复次数'] = df.groupby('ID')['ID'].transform('count')",
        "",
        "print(\"添加重复标记:\")",
        "print(df)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 技巧3: 合并重复数据的值",
        "",
        "对重复记录进行聚合（如求和、平均）。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "df_agg = pd.DataFrame({",
        "    'ID': [1, 1, 2, 2, 3],",
        "    '姓名': ['张三', '张三', '李四', '李四', '王五'],",
        "    '销售额': [1000, 1500, 2000, 1800, 3000]",
        "})",
        "",
        "print(\"原始数据:\")",
        "print(df_agg)",
        "",
        "print(\"\\n按 ID 聚合，销售额求和:\")",
        "df_sum = df_agg.groupby('ID', as_index=False).agg({",
        "    '姓名': 'first',  # 取第一个",
        "    '销售额': 'sum'  # 求和",
        "})",
        "print(df_sum)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 总结",
        "",
        "### 方法对比",
        "",
        "| 方法 | 功能 | 主要参数 | 适用场景 |",
        "|------|------|----------|----------|",
        "| `duplicated()` | 检测重复 | `subset`, `keep` | 探索重复情况 |",
        "| `drop_duplicates()` | 删除重复 | `subset`, `keep`, `inplace` | 清理重复数据 |",
        "",
        "### keep 参数说明",
        "",
        "| 值 | 含义 | 示例 |",
        "|----|------|------|",
        "| 'first' | 保留第一个，删除后续 | [1,1,1] → [1] |",
        "| 'last' | 保留最后一个，删除前面的 | [1,1,1] → [1] (最后一个) |",
        "| False | 删除所有重复 | [1,1,1] → [] |",
        "",
        "### 关键要点",
        "",
        "1. **先检测后删除**：使用 `duplicated()` 了解重复情况",
        "2. **选择保留策略**：根据业务需求选择 `keep` 参数",
        "3. **按列去重**：使用 `subset` 指定检查的列",
        "4. **处理部分重复**：根据业务逻辑决定是否删除",
        "5. **保留最新数据**：结合排序和 `keep='last'` 保留最新记录",
        "6. **聚合重复数据**：使用 `groupby` 合并重复记录的值",
        "7. **标记不删除**：添加标记列，保留重复信息"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.8.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}